MEASUREMENT SYSTEM 



[0001] This application Is based on Japanese Patent 
Application No. 2003-068290 filed on March 13, 2003, the 
contents of which are hereby Incorporated by reference. 

BACKGROUND OF THE INVENTION 

1 . Field of the Invention 

[0002] The present Invention relates to a measurement 
system for measuring an object based on Images obtained by 
plural cameras . 

2. Description of the Prior Art 

[0003] Conventionally, various techniques of stereoscopic 
measurement using plural cameras are proposed. For 
example, there Is proposed a technique In which two 
cameras are used to determine a distance away from an 
object to switch between a low velocity /high accuracy mode 
and a high velocity/ low accuracy mode depending on the 
distance (Japanese unexamined patent publication No. 8- 
219774). Further, there Is also proposed a technique In 
which two cameras are used to photograph an object to 
detect an intruder by the stereoscopic measurement 
(Japanese unexamined patent publication No. 2000-115810). 
[0004] In recent years, application of the stereoscopic 
measurement to a real-time system such as a robot or a 
monitoring system has been expected with improvement in 
quality of an image pickup device (an image sensor) and a 
processor and price -reduction thereof. 

[0005] One problem in the application of the stereoscopic 
measurement to a real-time system is how a trade-off 
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SUMMARY OF THE INVENTION 
[0008] It is an object of the present Invention to 
provide a measurement system that can conduct optimum 
measurement depending on environmental changes or movement 
of an object in a robot or a monitoring system. 
[0009] According to one aspect of the present invention, 
a measurement system for measuring an object based on 
images obtained by plural cameras includes a 
positional control portion for controlling positions of 
the cameras to change photographing directions of the 
cameras, a two-dimensional measurement portion for 
conducting two-dimensional measurement of the object based 
on the image of the object, the image being obtained by at 
least one of the cameras a stereoscopic measurement 
portion for conducting stereoscopic measurement of the 
object based on the images of the object, the images being 
obtained by the cameras, and a switching portion for 
switching between the two-dimensional measurement portion 
and the stereoscopic measurement portion to perform an 
operation* 

[0010] The positions of the ceuneras may be individually 
controlled. Alternatively, the positional relationship 
between the cameras may be fixed, and, the two cameras are 
made a group of cameras for performing position control. 
[0011] Preferably, the positional control portion 
controls the positions of the cameras so that the cameras 
photograph ranges differing from each other and face 
directions differing from each other when the two- 
dimensional measurement portion conducts two-dimensional 



measurement, and controls the positions of the cameras so 
that the cameras photograph an overlapping range when the 
stereoscopic measurement portion conducts stereoscopic 
measurement, the overlapping range Including the object, 
and the switching portion switches to operate the two- 
dimensional measurement portion In an Initial condition, 
and switches to operate the stereoscopic measurement 
portion when the two-dimensional measurement portion 
detects a moving object. 

[0012] Further, the stereoscopic measurement portion 
Includes a portion for reducing resolution of the Images, 
and switches between generation of three-dimensional data 
with high resolution and generation of three-dimensional 
data with low resolution appropriately to conduct 
stereoscopic measurement • 

[0013] Furthermore, each of the cameras Includes an Image 
pickup device In which a color filter having any one of 
three primary colors Is arranged for each pixel, and when 
Image data obtained by the cameras are processed. Image 
data of pixels corresponding to only a color filter with a 
particular color In the Image pickup device of each of the 
cameras are used. 

[0014] These and other characteristics and objects of the 
present Invention will become more apparent by the 
following descriptions of embodiments with reference to 
drawings • 

BRIEF DESCRIPTION OF THE DRAWING 
[0015] Fig. 1 shows a structure of a monitoring system 
according to a first embodiment of the present Invention. 



[0016] Fig. 2 is a block diagram showing an example of a 
structure of a two-dimensional processing portion. 
[0017] Fig. 3 is a block diagram showing an example of a 
structure of a stereo processing portion. 

[0018] Fig. 4 is a block diagram showing a structure of a 
modified two-dimensional processing portion. 
[0019] Fig. 5 is a block diagram showing a structure of a 
modified stereo processing portion. 

[0020] Fig. 6 shows a structure of a robot control system 
according to a second embodiment of the present invention. 
[0021] Fig. 7 is a block diagram showing a portion of an 
image input circuit. 

[0022] Fig. 8 shows a part of pixels of an image pickup 
device . 

[0023] Fig. 9 shows an example of a state in which a 
signal is allocated to each pixel. 



DESCRIPTION OF THE PREFERRED EMBODIMENTS 
[ First Embodiment ] 

[0024] An application example of a measurement system 
according to the present invention to a monitoring system 
for security is explained below. 

[0025] Fig. 1 shows a structure of a monitoring system 1 
according to a first embodiment of the present invention. 
Fig. 2 is a block diagram showing an example of a 
structure of a two-dimensional processing portion 41, Fig. 
3 is a block diagram showing an example of a structure of 
a stereo processing portion 42, Fig. 4 is a block diagram 
showing a structure of a modified two-dimensional 
processing portion 4 IB and Fig. 5 is a block diagram 



showing a structure of a modified stereo processing 
portion 42B. 

[0026] As shown in Fig. 1, the monitoring system 1 
includes two cameras 11 and 21, a pan mechanism 12 and a 
tilt mechanism 13 that are used for changing the 
photographing direction of the camera 11, a pan mechanism 
22 and a tilt mechanism 23 that are used for changing the 
photographing direction of the camera 21, a driver 14 for 
controlling the pan mechanism 12 and the tilt mechanism 13, 
a driver 24 for controlling the pan mechanism 22 and the 
tilt mechanism 23, positional control mechanisms 31 and 32, 
a driver 33 for controlling the positional control 
mechanisms 31 and 32, a two-dimensional processing portion 
41, a stereo processing portion 42, a controller 43 and an 
output portion 44. 

[0027] Each of the cameras 11 and 21 includes an optical 
system, an image pickup device, a zoom mechanism and a 
drive circuit therefor. Each of the cameras 11 and 21 
photographs an area with a predetermined range (an area to 
be shot) depending on a zooming operation using the image 
pickup device. The shot image may be an image of a 
background within the area to be shot or an image of an 
object to be shot (an object). Data of one frame are 
output out of the obtained image data at an appropriate 
cycle. For example, 30 frames as image data per second 
are output. Further, a signal from outside enables control 
of a zooming operation or others. The structure and the 
operation of each of the cameras 11 and 21 per se are 
conventionally known. 
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[0028] The pan mechanisms 12 and 22 rotate the cameras 11 
and 21 from side to side respectively, thereby leading to 
the swing of the optical axis of each of the cameras 11 
and 21 from side to side. The tilt mechanisms 13 and 23 
rotate the cameras 11 and 21 up and down respectively, 
thereby leading to the swing of the optical axis of each 
of the cameras 11 and 21 up and down. The drivers 14 and 
24 control drive of the pan mechanisms 12 and 22 as well 
as the tilt mechanisms 13 and 23 based on command signals 
from the controller 43. 

[0029] The positional control mechanisms 31 and 32 
control the entire position and posture of the cameras 11 
and 21 Including the pan mechanisms 12 and 22 and the tilt 
mechanisms 13 and 23. Stated differently, the operation 
of the positional control mechanisms 31 and 32 changes the 
entire position and posture of the cameras 11 and 21 with 
the positional relationship between the cameras 11 and 21 
being maintained. The driver 33 controls drive of the 
positional control mechanisms 31 and 32 based on command 
signals from the controller 43. 

[0030] The cameras 11 and 21, and the positional control 
mechanism therefor are so Installed that a target area for 
monitoring In an entrance of a building, an entrance of a 
room, a corridor, a lobby, a reception desk or a warehouse 
Is Included In an angle of view. 

[0031] Based on each of the Images (Image data) Dl and D2 
obtained by the cameras 11 and 21, the two-dimensional 
processing portion 41 performs processing for two- 
dimensional measurement of the object Individually to 
output measurement data D3 . 
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[0032] Referring to Fig. 2, the two-dimensional 
processing portion 41 includes a one-frame delay portion 
411 and a two-dimensional movement detection portion 412, 
The one-frame delay portion 411 memorizes one frame of the 
image Dl or D2 to output the memorized image Dl or D2 with 
one frame being delayed. The two-dimensional movement 
detection portion 412 compares the image Dl or D2 of the 
current frame with image DIT or D2T in which one freune is 
delayed, then to detect the object based on a change seen 
in the comparison result. 

[0033] As a technique for detecting an object, for 
example, it is possible to employ a well-known technique 
such as a technique using background subtraction, a 
technique using time subtraction or a technique using 
movement vectors of time-series images (an optical flow 
technique). In the case of the technique using time 
subtraction, for example, a subtraction image between the 
current frame image and the previous frame image is 
derived by computing, and preliminary judgment is made 
that an object is present when the sum of intensity of the 
subtraction images is a threshold level or more. On this 
occasion, the processing is simple and high-speed 
processing is possible. However, there is a possibility 
that even variation in illumination, i.e., brightness in 
the frame, the presence or absence of shadow and size of 
the shadow may be detected as an object. 
[0034] Based on the images Dl and D2 obtained by the 
cameras 11 and 21, the stereo processing portion 42 
performs processing for stereoscopic measurement of the 
object to output measurement data D4. 



[0035] Referring to Fig. 3, the stereo processing portion 
42 includes a stereo image processing portion 421, a one- 
frame delay portion 422 and a three-dimensional movement 
detection portion 423. The stereo image processing 
portion 421 generates a distance image (three-dimensional 
data) DT based on the two images Dl and D2 using the 
triangulation principle. The one-frame delay portion 422 
memorizes one frame of the distance image DT to output the 
memorized distance image DT with one frame being delayed. 
The three-dimensional movement detection portion 423 
compares the distance image DT output from the stereo 
image processing portion 421 with a distance image DTT in 
which one frame is delayed, then to detect the detailed 
status of the object based on a change seen in the 
comparison result. 

[0036] More specifically, one of the cameras 11 and 21 is 
made a reference camera and the other is made a referred 
camera. The stereo image processing portion 421 searches 
corresponding points between the image Dl taken by the 
reference camera (reference image) and the image D2 taken 
by the referred camera (referred image). The distance 
image DT is calculated in connection with each pixel in 
the reference image based on optical parameters corrected 
beforehand and the positional relationship between the two 
cameras. In this case, since the processing is 
complicated, the processing rate is low. However, there 
is little possibility that variation in illumination may 
affect detection of an object. 

[0037] The three-dimensional movement detection portion 
423 derives a subtraction distance image between the 
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distance image DT of the current frame and the distance 
image DTT of the previous frame by computing, and 
definitive judgment is rendered that an object is actually 
present when the sum of intensity of the subtraction 
distance images is a threshold level or more. 
[0038] In the case of using the monitoring system 1 for 
security, the two-dimensional processing portion 41 
detects, for example, an intruder as the object based on 
the two images Dl and D2 obtained by shooting ranges 
differing from each other. In accordance with the 
position and posture of each of the cameras 11 and 21, and 
the position and size of the object seen in the images Dl 
and D2, the two-dimensional processing portion 41 outputs 
information of a rough position of the intruder and rough 
size thereof to the controller 43 as the measurement data 
D3. The controller 43 controls the position, the posture 
and the zooming operation of each of the cameras 11 and 21 
so that the intruder can be zoomed in. The stereo 
processing portion 42 conducts three-dimensional 
measurement based on the images Dl and D2, then to output 
information indicative of the position of the intruder, 
i.e., the distance away from the intruder, and the size of 
the intruder to the controller 43 as the measurement data 
D4. 

[0039] The measurement data D3 include the images Dl and 
D2. The measurement data D4 include the distance image DT. 
The measurement data D4 are used to judge accurately 
whether the intruder detected as the object by the two- 
dimensional processing portion 41 is actually an intruder. 
[0040] Various known algorithms are used for decision 



with respect to criteria for determining that the object 
is an intruder, i.e., intensity of a subtraction image and 
of a subtraction distance image. 

[0041] As described above, the controller 43 controls the 
posture of each of the cameras 11 and 21 from side to side 
and up and down. Further, the controller 43 switches the 
setting that the images Dl and D2 taken by the cameras 11 
and 21 are processed by the two-dimensional processing 
portion 41 or by the stereo processing portion 42. 
[0042] Generally, the position of each of the cameras 11 
and 21 is so controlled that each of the cameras 11 and 21 
shoots a different range and faces a different direction, 
and each of the cameras 11 and 21 is so controlled that 
wide-angle zooming is achieved. In this case, the 
boundary portion between the images Dl and D2 taken by the 
cameras 11 and 21 may somewhat overlap each other. Thus, 
the cameras 11 and 21 shoot a wide range. During the 
period when each of the cameras 11 and 21 shoots a 
different range, the controller 43 switches the setting so 
that the images Dl and D2 are processed by the two- 
dimensional processing portion 41. Additionally, the 
cameras 11 and 21 may be moved so as to scan around, so 
that a wider range is photographed. 
[0043] When an intruder is detected, for example, 
position control and zooming control of each of the 
cameras 11 and 21 are so performed that both the cameras 
11 and 21 magnify the intruder for photographing the same. 
Stated differently, both the cameras 11 and 21 photograph 
ranges including the intruder, the ranges being overlapped 
with each other. The controller 43 switches the setting 



so that the Images Dl and D2 are processed by the stereo 
processing portion 42. 

[0044] More specifically, the controller 43 has mode 
Information therein for controlling two modes. I.e., a 
two-dimensional measurement mode (a monocular measurement 
mode) and a stereoscopic measurement mode. The controller 
43 switches the setting so that the Images Dl and D2 are 
processed by the two-dimensional processing portion 41 or 
the stereo processing portion 42 based on the measurement 
data D3 output from the two-dimensional movement detection 
portion 412, the measurement data D4 output from the 
three-dimensional movement detection portion 423 and the 
mode Information. The switching allows the setting of the 
two-dimensional measurement mode or the stereoscopic 
measurement mode. The controller 43 outputs a switching 
signal DC depending on the mode. As the switching signal 
DC, for example, the controller 43 outputs an OFF signal 
for the two-dimensional measurement mode and an ON signal 
for the stereoscopic measurement mode. The switching 
signal DC switches the presence or absence of the 
operation of the two-dimensional processing portion 41 and 
the stereo processing portion 42. Further, the switching 
signal DC may be used for switching an output destination 
of the Images Dl and D2, an output destination of each 
block and the presence or absence of an operation of each 
block . 

[0045] The controller 43 further outputs an alarm signal 
D5 for notifying that an Intruder Is detected. In 
accordance with the measurement data D3 or D4 output from 
the two-dimensional processing portion 41 or the stereo 
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processlng portion 42. Further, when the controller 43 
switches from processing by the two-dimensional processing 
portion 41 to processing by the stereo processing portion 
42, the controller 43 may output the alarm signal D5 to 
raise an alarm. 

[0046] Based on the alarm signal D5, the output portion 
44 notifies an observer that an Intruder Is detected by 
audio or Image display. 

[0047] Additionally, the controller 43 or the output 
portion 44 Is so structured that the same can communicate 
with an external host computer or an external terminal via 
a LAN or other networks. The communication enables the 
images Dl and D2, and the measurement data D3 and D4 to be 
output to the host computer. 

[0048] When an intruder is detected in the stereo 
processing portion 42, for example, the distance image DT 
and the reference image Dl are output. Vftien no intruder 
is detected in the stereo processing portion 42, only the 
reference image Dl is output together with time 
information . 

[0049] The positional control mechanisms 31 and 32 are 
used in addition to the pan mechanisms 12 and 22 as well 
as the tilt mechanisms 13 and 23 for control of the 
position or posture of each of the cameras 11 and 21. For 
example, when one of the cameras 11 and 21 detects an 
intruder, the pan mechanisms 12 and 22, the tilt 
mechanisms 13 and 23 and the positional control mechanisms 
31 and 32 are so controlled appropriately that the other 
camera faces the intruder with the posture of the ccimera 
detecting the intruder being controlled so as to chase and 



photograph the intruder. On this occasion, position 
control is so performed that the base line connecting the 
two cameras 11 and 21 becomes perpendicular to the 
direction of the intruder in the end. At the time point 
when the base line becomes perpendicular to the direction 
of the intruder, the setting may be switched to the 
stereoscopic measurement mode. Thereby, the base length 
of each of the cameras 11 and 21 for the intruder is 
maximized so that the intruder can be photographed with 
large parallax, resulting in the stereoscopic measurement 
with higher degree of accuracy. 

[0050] In control of the position or posture of each of 
the cameras 11 and 21, control may be so performed that 
the pan mechanisms 12 and 22 make the cameras 11 and 21 
move symmetrically, and the tilt mechanisms 13 and 23 make 
the cameras 11 and 21 move synchronously. Thereby the 
mechanisms are simplified and the control is facilitated, 
leading to the simplified processing in the two- 
dimensional processing portion 41 and the stereo 
processing portion 42. 

[0051] As described above, in the case of the control of 
the position or posture of each of the cameras 11 and 21, 
positioning is performed by the control using the pan 
mechanisms 12 and 22, the tilt mechanisms 13 and 23 and 
the positional control mechanisms 31 and 32. However, 
the monitoring system 1 may be so structured that 
positioning of each of the cameras 11 and 21 is performed 
mechanically at an appropriate position. The mechanical 
positioning is so performed that the appropriate position 
is, for example, a position where the optical axes of the 
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cameras 11 and 21 are parallel to each other, a position 
where the cameras 11 and 21 are made both ends of a base 
of an isosceles triangle whose vertex is an object within 
a specific distance range, a position where a specific 
object to be monitored constantly is shot or others. A 
stopper or a notch can be used for the mechanical 
positioning, for example. The mechanical positioning 
mentioned above improves positional accuracy and enhances 
measurement accuracy without positioning control with high 
degree of precision. 

[0052] Next, the flow of the entire operation of the 
monitoring system 1 is described. 

[0053] (1) First, when the power source of the monitoring 
system 1 is turned on, the mode information inside the 
controller 43 is initialized to the two-dimensional 
measurement mode. Accordingly, in the initial condition, 
the controller 43 outputs an OFF switching signal DC in 
order to set the monitoring system 1 to the two- 
dimensional measurement mode. 

[0054] In the embodiment described above, the images Dl 
and D2 are processed individually in the two-dimensional 
processing portion 41. However, when the cameras 11 and 
21 are set to photograph the same range, two-dimensional 
measurement of an object may be carried out in the two- 
dimensional processing portion 41 using only one of the 
Images, for example, the reference image Dl. Thereby, the 
processing in the two-dimensional processing portion 41 is 
further facilitated. 

[0055] (2) Next, when the previous frame image is input to 
the two-dimensional processing portion 41, the two- 
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dlmenslonal movement detection portion 412 judges whether 
or not an object moves in the scene using the current 
frame image and the previous frame image to output the 
decision result to the controller 43. In this case, as 
mentioned above, there is a possibility that even 
variation in illumination may be detected as movement of 
the object. 

[0056] (3) When the reference image Dl is input to the 
stereo processing portion 42, the distance image DT is 
output to the three-dimensional movement detection portion 
423 from the stereo processing portion 42. The three- 
dimensional movement detection portion 423 judges whether 
or not an object moves in the scene using the distance 
image DT of the current frame and the distance image DTT 
of the previous frame to output the decision result to the 
controller 43. 

[0057] (4) When the mode information is the two- 
dimensional measurement mode, the controller 43 changes 
the mode information to the stereoscopic measurement mode 
in response to output of the measurement data D3 from the 
two-dimensional movement detection portion 412, the 
measurement data D3 indicating the presence of movement of 
the object. Then, the controller 43 switches so that the 
images Dl and D2 are processed in the stereo processing 
portion 42. The mode information is maintained as the 
two-dimensional measurement mode until the measurement 
data D3 are output from the two-dimensional movement 
detection portion 412, the measurement data D3 indicating 
the presence of movement of the object. 
[0058] When the mode information is the stereoscopic 



measurement mode, the controller 43 changes the mode 
information to the two-dimensional measurement mode in 
response to output of the measurement data D4 from the 
three-dimensional movement detection portion 423, the 
measurement data D4 indicating the absence of movement of 
the object. Then, the controller 43 switches so that the 
images Dl and D2 are processed in the two-dimensional 
processing portion 41. When the measurement data D4 
indicating the presence of movement of the object are 
output from the three-dimensional movement detection 
portion 423, the mode information is maintained as the 
stereoscopic measurement mode and the distance image DT is 
output to the output portion 44. 

[0059] (5) The output portion 44 transmits the reference 
image Dl to the host computer along with the time 
information at regular intervals . When the distance image 
DT is output, the distance image DT is transmitted along 
with the reference image Dl of the same time as the 
distance image DT. 

[0060] (6) When the host computer receives the reference 
image Dl or the distance image DT, the host computer 
records the same along with time. When the distance image 
DT is transmitted, the host computer raises an alarm 
simultaneous with recording - 

[0061] According to the embodiment described above, the 
movement of the object is detected by comparing the image 
Dl or the distance image DT of the current frame and the 
image DIT or the distance image DTT of the previous frame. 
However, as shown in Figs. 4 and 5, it is possible to 
compare the image Dl or the distance image DT of the 
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current frame and an image (a background image) DIN or a 
distance image DTN in the initial condition or at reset. 
In this case, an initial image memorizing portion 41 IB and 
an initial distance image memorizing portion 422B are 
provided in lieu of the one -frame delay portion 411 and 
the one-frame delay portion 422. 

[0062] The connection or operation of each of the blocks 
may be so controlled that the initial image memorizing 
portion 41 IB memorizes the image DIN in the initial 
condition, and the initial distance image memorizing 
portion 422B memorizes the distance image DTN in the 
initial condition. 

[0063] More specifically, for example, when the 
monitoring system 1 starts, the controller 43 outputs a 
reset signal so that the reference image Dl is input to 
the initial image memorizing portion 41 IB. At the same 
time, the reference image Dl is memorized as the reference 
image DIN in the initial image memorizing portion 411B. 
Further, the stereo processing portion 42 is caused to 
generate the distance image DT based on the images Dl and 
D2 and the generated initial distance image DTN is 
memorized in the initial distance image memorizing portion 
422B. While the monitoring system 1 is active, movement 
of the object is detected using the image DIN in the 
initial condition and the distance image DTN in the 
initial condition that are memorized individually. 
[Second Embodiment] 

[0064] Next, an application example of a measurement 
system according to the present invention to a robot 
navigation system is explained below. 



[0065] Fig. 6 shows a structure of a robot control system 
2 according to a second embodiment of the present 
invention . 

[0066] The robot control system 2 according to the second 
embodiment is installed inside a robot. The robot is 
movable back and forth from side to side by control of the 
robot control system 2. Moreover, the head of the robot 
is provided with a stereoscopic camera having a pan/ tilt 
mechanism, and the stereoscopic camera operates in 
accordance with a command of the robot control system 2 
inside the robot. 

[0067] Here, the stereoscopic camera and the pan/tilt 
mechanism may be similar to the cameras 11 and 21, and the 
position and posture control mechanism in the first 
embodiment. Alternatively, the cameras 11 and 21 and the 
position and posture control mechanism may be simplified 
for use. The cameras 11 and 21 and others are omitted in 
Fig. 6. A driver for the position and posture control 
mechanism is shown as a pan/tilt control portion 61. 
[0068] Referring to Fig. 6, the robot control system 2 
includes resolution lowering portions 51 and 52, a stereo 
processing portion 53, a three-dimensional matching 
portion 54, a position identification portion 55, a three- 
dimensional map update portion 56, a three— dimensional map 
memorizing portion 57, a position and posture memorizing 
portion 58, a controller 59, a motion control portion 60 
and the pan/tilt control portion 61. 

[0069] The resolution lowering portion 51 or 52 reduces 
resolution of an image Dl or D2 output from the camera 11 
or 21 to output a low resolution image DIL or D2L in which 
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the total number of pixels is reduced. For example, image 
data are subtracted to reduce the resolution of the image 
to a half, one third or one fourth, then to output an 
image in which the image data are reduced correspondingly. 
[0070] Similar to the case of the first embodiment, the 
stereo processing portion 53 performs processing for 
stereoscopic measurement of an object based on the images 
Dl and D2 or the lower resolution images DIL and D2L 
thereof to output measurement data D4 including a distance 
image DT. 

[0071] The three-dimensional matching portion 54 checks 
the distance image (partial three-dimensional data) DT 
output from the stereo processing portion 53 against a 
three-dimensional map DM previously memorized in the 
three— dimensional map memorizing portion 57. Stated 
differently, matching is performed between the distance 
image DT the robot see via the cameras 11 and 21 and the 
three-dimensional map DM. Then, a part of the three- 
dimensional map DM corresponding to the distance image DT 
is detected to output position and posture information D6 
of the distance image DT corresponding to the three- 
dimensional map DM. In the case of the check, the three- 
dimensional matching portion 54 outputs a check error 
signal DE to the controller 59 when the degree of the 
check is lower than a threshold level. 

[0072] The position identification portion 55 computes a 
position and posture of the robot based on the position 
and posture information D6 output from the three- 
dimensional matching portion 54 and position and posture 
information of the cameras 11 and 21 to output position 
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and posture information D7. The position and posture 
information of the cameras 11 and 21 is obtained based on 
information of the pan/tilt control portion 61. 
[0073] The three-dimensional map update portion 56 
replaces the distance image DT output from the stereo 
processing portion 53 with the corresponding part of the 
three-dimensional map DM. Thereby, the three-dimensional 
map DM memorized in the three— dimensional map memorizing 
portion 57 is updated. 

[0074] The controller 59 serves as a central controller. 
More specifically, the controller 59 manages tasks of the 
robot and controls each portion of the robot based on the 
tasks. The controller 59 computes a movement path of the 
robot in accordance with the contents of the tasks, and 
receives necessary information from the cameras 11 and 21 
appropriately, and issues a command to the motion control 
portion 60, the command being for forcing the robot to go 
the computed path. 

[0075] Further, the controller 59 outputs a mode signal 
DD for switching between a high velocity mode and a high 
accuracy mode, vntien the mode signal DD is ON, the mode is 
switched to the high velocity mode and the image Dl or D2 
is input to the resolution lowering portion 51 or 52, and 
the output from the stereo processing portion 53 is input 
to the three-dimensional matching portion 54. 
[0076] The motion control portion 60 controls drive of 
wheels to control movement and turn of the robot . 
[0077] The pan/tilt control portion 61 controls the line- 
of- sight direction of each of the cameras 11 and 21 
responding to a command from the controller 59. On this 
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occasion, posture Information of each of the cameras 11 
and 21 Is output occasionally. 

[0078] Next, the flow of the entire operation of the 
robot control system 2 Is described. 

[0079] (1) First, when the power source of the robot 
control system 2 Is turned on, the controller 59 outputs 
an OFF mode signal DD to set the robot control system 2 to 
the high accuracy mode. The robot Inputs a plurality of 
Images Dl and D2 while scanning around with the cameras 11 
and 21 controlled by the pan /tilt mechanism with the robot 
remaining stationary. Based on the plural Images Dl and 
D2, a plurality of distance Images DT having high degree 
of accuracy Is generated. The distance Images DT are used 
to prepare a three-dimensional map DM. 

[0080] In the high accuracy mode, the two Images Dl and 
D2 are Input to the stereo processing portion 53 without 
passing through the resolution lowering portions 51 and 52, 
respectively. Thereby, a distance Image DT having higher 
resolution and higher degree of accuracy Is generated 
compared to the case where the Images Dl and d2 are passed 
through the resolution lowering portions 51 and 52, 
respectively. However, the computing cost Increases, 
leading to a low processing rate. 

[0081] (2) vnien the robot starts to move, the controller 
59 outputs an ON mode signal to switch the mode to the 
high velocity mode. The position and posture of the robot 
are calculated by checking the generated distance Image DT 
against the three-dimensional map DM memorized In the 
three-dimensional map memorizing portion 57. When the 
robot Is out of the predetermined path, the controller 59 
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instructs correction movement to the motion control 
portion 60, the correction movement being for forcing the 
robot to the predetermined path, 

[0082] In the high velocity mode, the two images Dl and 
D2 are passed through the resolution lowering portions 51 
and 52 respectively to input to the stereo processing 
portion 53. Thereby, a distance image DT having lower 
resolution and lower degree of accuracy is generated 
compared to the case where neither the image Dl nor the 
image D2 is passed through the resolution lowering 
portions 51 and 52. However, the computing cost reduces, 
leading to a high processing rate. 

[0083] (3) Under the case of (2) mentioned above, when 
the controller 59 detects that the three-dimensional 
matching portion 54 outputs a check error signal DE, the 
controller 59 judges that an abnormality occurs or an 
environment around the robot changes, and issues an 
instruction to the motion control portion 60 for letting 
the robot stationary. Then, the controller 59 switches 
the mode to the high accuracy mode to perform processing 
similar to the case of (1) mentioned above for 
restructuring the three-dimensional map DM. 
[Modification Example] 

[0084] Next, a modification of each of the embodiments 
mentioned above is described with respect to a circuit for 
reading out the images Dl and D2 output from the cameras 
11 and 21. 

[0085] Fig. 7 is a block diagram showing a portion of an 
image input circuit. Fig. 8 shows a part of pixels of an 
image pickup device and Fig. 9 shows an example of a state 
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in which a signal is allocated to each pixel. 
[0086] A color CCD is commonly used as an image pickup 
device in each of the cameras 11 and 21. An inexpensive 
camera often has a CCD in which a color filter having any 
one of three primary colors of red, green and blue is 
applied to each pixel. Such an entire color filter 
sometimes may be referred to as a color mosaic filter. 
The typical example of the color mosaic filter is an RGB 
Bayer filter FLl shown in Fig. 8. The use of the RGB 
Bayer filter FLl permits representation of one pixel of 
each of the images Dl and D2 of an object by means of four 
pixels including two green pixels, one red pixel and one 
blue pixel. 

[0087] Meanwhile, it is economical that a color image is 
used as an image for recording, and a luminance image is 
used as an image for stereoscopic measurement. The reason 
is that the use of luminance components is the most 
effective for correspondence in the stereoscopic 
measurement, and the use of color components increases a 
computing cost with low efficiency. 

[0088] Therefore, when measurement is conducted by the 
stereoscopic measurement mode, luminance components are 
extracted from the CCD of each of the cameras 11 and 21 to 
be interlaced for each pixel, then to be output. In order 
to realize this operation, the circuit is provided with a 
one -pixel delay portion 71 for delaying one pixel of the 
output from the camera 11 and a pixel synchronization 
control portion 72 for controlling the pixels 
synchronously . 

[0089] More specifically, the one-pixel delay portion 71 



delays one pixel of an RAW linage whose pixel data are 
serially output from the CCD of the camera 11 as a 
referred camera In raster order, the RAW Image having a 
Bayer arrangement. The switch SW Is so switched that a 
green pixel Is output from the camera 11 at the timing 
when a red pixel or a blue pixel Is output from the CCD of 
the camera 21 as a reference camera. The pixel 
synchronization control portion 72 controls timing for the 
switching. As shown In Fig. 9, the switch SW outputs 
Image signals from the CCD of the camera 11 and Image 
signals from the CCD of the camera 21 alternately. The 
output Image signals are quantized using one A/D converter 
[0090] Referring to Fig. 9, Image components of green 
(GS) of the reference camera 21 and Image components of 
green (GR) of the referred camera 11 are output 
alternately . 

[0091] Thereby, It Is possible to capture an Image 12 In 
which the green pixel components of the cameras 11 and 21 
are arranged alternately by a single system line. The 
captured Image D12 Is memorized In an appropriate memory. 
The designation of an address allows the two Images Dl and 
D2 to be read out separately. Thus, Image data of pixels 
corresponding to only a color filter with a particular 
color In the CCD of each of the cameras 11 and 21 are used 
[0092] Thereby, capture of Images from the cameras 11 and 
21 Is speeded up In the stereoscopic measurement mode. 
Further, such a structure permits a color Image (a Bayer 
arrangement RAW Image) of the reference Ccunera 21 to be 
read out of the CCD without any change In the two- 
dimensional measurement mode. 



[0093] According to the embodiments described above. In a 
robot and a monitoring system, an environmental change and 
object movement are triggered to switch between a two- 
dimensional measurement mode and a stereoscopic 
measurement mode, and therefore optimum measurement can be 
conducted with a low cost structure. 

[0094] Further, position control Is so performed that 
photograph ranges of the cameras 11 and 21 are different 
from each other. Then, position control Is so performed 
that both the cameras 11 and 21 conduct stereoscopic 
measurement when an Intruder or the like Is detected. 
Thereby, it Is possible to achieve wide-ranging monitoring 
and determination of the presence or absence of an 
Intruder with high degree of accuracy. 

[0095] According to the embodiments described above, the 
positions of the cameras 11 and 21 are mainly controlled 
by the pan mechanisms 12 and 22 as well as the tilt 
mechanisms 13 and 23, respectively. However, the relative 
positional relationship between the cameras 11 and 21 may 
be fixed. On this occasion, the positional relationship 
between the ceimeras 11 and 21 may be fixed, and besides, 
the positional control mechanisms 31 and 32 or others may 
control the position of the whole of the cameras 11 and 21. 
For ex£imple, in the robot control system 2 according to 
the second embodiment, the positional relationship between 
the two cameras is fixed. The two cameras are made a 
group of cameras, the whole of which is controlled so as 
to be panned and tilted. The mode is switched between the 
two-dimensional measurement mode and the stereoscopic 
measurement mode, similar to the case of the first 



embodiment. In this case, only an Image photographed by 
one of the cameras Is used for two-dimensional measurement. 
However, two Images photographed by both the cameras can 
be used as two-dimensional Images. In the case of 
stereoscopic measurement, correspondence between an Image 
photographed by one of the cameras and an Image 
photographed by the other Is conducted. 

[0096] In the embodiments described above, the cameras 11 
and 21 are placed In the lateral direction (the horizontal 
direction) slde-by-slde. However, the cameras may be 
placed In the longitudinal direction (the vertical 
direction) or placed diagonally. It Is also possible to 
use three or more cameras . Each portion of the monitoring 
system 1 or the robot control system 2 can be realized 
using a CPU, a memory and others In terms of software, or 
using a hardware circuit or In combination thereof. 
[0097] In the foregoing embodiments, structures, 
circuits, shapes, dimensions, numbers and processing 
contents of each part or whole part of the monitoring 
system 1 or the robot control system 2 can be varied as 
required within the scope of the present Invention. The 
present Invention can be used for various applications 
other than the monitoring system and the robot control 
system. 



